A light-weight package providing Julia users easy access to the Trillion Dollar Words dataset and model (Shah, Paturi, and Chava 2023).
Disclaimer
Please note that I am not the author of the Trillion Dollar Words paper nor am I affiliated with the authors. The package was developed as a by-product of our research and is not officially endorsed by the authors of the paper.
We have archived activations for each layer and sentence as artifacts:
usingLazyArtifactsartifact"activations_layer_24"
OK, but why would I need all this? 🤔
“There! It’s sentient!”
Motivation
: „It is essential to bring inflation back to target to avoid drifting into deflation territory.“
: „It is essential to bring the numbers of doves back to target to avoid drifting into dovelation territory.“
Motivation
: „It is essential to bring inflation back to target to avoid drifting into deflation territory.“
: „It is essential to bring the numbers of doves back to target to avoid drifting into dovelation territory.“
“They’re exactly the same.”
— Linear probe
Embedding FOMC comms
We linearly probe all layers to predict unseen economic indicators (CPI, PPI, UST yields).
Predictive power increases with layer depth and probes outperform simple AR() models.
Figure 1: Out-of-sample root mean squared error (RMSE) for the linear probe plotted against FOMC-RoBERTa’s n-th layer for different indicators.
Sparks of Economic Understanding?
If probe results were indicative of some intrinsic ‘understanding’ of the economy, then the probe should not be sensitive to random sentences unrelated to economics.
Figure 2: Probe predictions for sentences about inflation of prices (IP), deflation of prices (DP), inflation of birds (IB) and deflation of birds (DB). The vertical axis shows predicted inflation levels subtracted by the average predicted value of the probe for random noise.
Intended Purpose and Goals
Good starting point for the following ideas:
Fine-tune additional models on the classification task or other tasks of interest.
Further model probing, e.g. using other market indicators not discussed in the original paper.
Improve and extend the label annotations.
Any contributions are very much welcome.
Questions?
With thanks to my co-authors Andrew M. Demetriou, Antony Bartlett, and Cynthia C. S. Liem and to the audience for their attention.
References
Altmeyer, Patrick, Andrew M. Demetriou, Antony Bartlett, and Cynthia C. S. Liem. 2024. “Position: Stop Making Unscientific AGI Performance Claims.”https://arxiv.org/abs/2402.03962.
Shah, Agam, Suvan Paturi, and Sudheer Chava. 2023. “Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis.”arXiv Preprint arXiv:2310.02207v1. https://arxiv.org/abs/2305.07972.
Image sources
Leonardo DiCaprio: Meme template by user on Reddit
Quote sources
“There! It’s sentient”—that engineer at Google (probably!)